Students' perceptions of Blackboard and Moodle in a Portuguese university

The use of learning management systems (LMS) has grown considerably in universities around the world. The University of Minho (UM) has pioneered in this area in Portugal, adopting Blackboard as its official LMS. Moodle is also used in UM in scattered initiatives, allowing for interesting comparisons. Previous studies comparing Blackboard and Moodle have been confined to limited samples and focused on students’ perceptions only. In this paper, we also try to relate those perceptions to the impact of the LMSs on student level of engagement. We assess the extent and depth of use of the two LMSs, presenting the results of a study of students’ perceptions and experience with both Blackboard and Moodle. Unlike previous studies, more students (46.5%) stated a preference for Blackboard over Moodle, while 34.7% preferred Moodle, and nearly 20% had no preference. Factors that might explain these results are explored in some detail. By and large, a basic utilisation is made of both platforms, as little more than electronic document repositories, in what Francis and Raftery, in 2005, designate as a Mode 1 level of engagement. We could, however, detect some instances of a more sophisticated Mode 2 utilisation, particularly with Moodle, underlining the role of faculty in integrating a sophisticated use of LMSs when designing their courses. The use of learning management systems (LMSs) in learning environments In recent years, we have witnessed the growing introduction of LMSs in all levels of schooling in Portugal. Establishments using LMSs are still a minority (a little above 25% of universities according to a study by DeltaConsultores, 2007), but this should evolve rapidly. The use of information technology (IT) in schools and universities has been strongly supported by the government through initiatives ranging from primary schools to higher education that have encouraged the acquisition of laptop computers for students and teachers under favourable conditions and secured broadband connections in all public establishments. Around 960 000 laptop computers have been purchased by secondary school students and teachers, and about 370 000 by primary schools (GCELPT, 2009). Students entering universities nowadays will already have their own laptop computer with an Internet connection, and a growing number will have used an LMS in school. British Journal of Educational Technology Vol 42 No 5 2011 824–841 doi:10.1111/j.1467-8535.2010.01097.x © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. Despite the increase in LMS adoption in schools and universities, concern has been expressed as to whether LMSs are being used as effective learning tools or merely as electronic document repositories (Badge, Cann & Scott, 2005; Hall, 2006). Although there are some data on the overall adoption of LMSs in Portuguese schools and universities, very little is known as to how LMSs are used in practice. For example, we are yet to establish the extent to which this adoption is anywhere close to ‘sustainable embedding’, where e-learning is seen as ‘an accepted and expected part of teaching and learning’ (Nichols, 2008, p. 603). Hall (2006) argues that successful implementations of LMSs depend, not only on providing training and support for instructors, but also on the level of student active engagement and student and instructor satisfaction with the LMS used. The user perspective is therefore crucial to examine the implementation of LMSs and to evaluate their success. Francis and Raftery (2005, p. 2) distinguish among three e-learning modes of engagement, which correspond to growing levels of complexity and depth in LMS usage. Mode 1 is labelled ‘baseline course administration and learner support’, and illustrates situations where an LMS is used only ‘to distribute course information and carry out course administration’ and most learning activities occur in traditional classroom settings. In Mode 2, there is ‘blended learning leading to significant enhancements to learning and teaching processes’. This mode entails combining face-to-face interaction with a more widespread use of features such as: communication tools promoting interaction between tutor and students or among students; collaboration tools that assist students in the development of group work and otherwise allow for the sharing of learning resources; assessment tools, including the submission of graded assignments, the completion of quiz tests and the provision of feedback on work submitted; and finally, the inclusion of learning content that allows each student to independently further their learning at their own pace and specific interest. The third mode corresponds to a fully-fledged ‘online course/module’ where most learning will take place using the LMS, with extensive use of the tools described above and only marginal face-to-face interaction. These aspects affect the interpretation of data on LMS adoption. Nichols (2008) reveals considerable diversity among institutions in their progress towards e-learning sustainability. The fact that a learning platform is available, or that it is used extensively, does not necessarily mean they are used to similar levels of engagement. According to Hall (2006), most universities are at stage 1 of the Francis and Raftery (2005) model. Any comparison between alternative LMSs must also take into account the level of engagement to which they are being used. In this paper, we seek to assess the extent and depth of the use of different LMSs in the University of Minho (Portugal). Our research question is ‘Which are the perceptions and experiences of students in their use of Blackboard and Moodle?’ After reviewing previous studies that compare Blackboard and Moodle and explaining the University of Minho’s setting, we describe our data collection process. We then present our results, subsequently discussing them and drawing conclusions that contribute to the research purpose stated above. Comparing Blackboard and Moodle One of the most popular LMSs in Portugal is Moodle, an open source platform that ‘educators can use to create effective online learning sites’ (Moodle, n.d.). In 2007, it was used by 14% of Portuguese universities and most basic and secondary schools (DeltaConsultores, 2007). Moodle has actually become the standard LMS in public primary and secondary schools, and is encouraged by governmental projects. Blackboard has preferentially targeted universities, being the world leading proprietary LMS. It is used by a few Portuguese higher education institutions. Apart from the University of Minho, it has been adopted by five other universities and some polytechnic institutes. Student Perceptions of Blackboard and Moodle 825 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. Comparing these two LMSs is particularly interesting, not only because they are some of the more widely used platforms, but also because they present different features and functionalities that might be more or less adequate for different learning settings and objectives. Moreover, they represent quite different financial commitments to the institutions using them. We should not naïvely expect Moodle to be cost-free just because it is free to acquire and use. Administration and maintenance costs must be considered (Bremer & Bryant, 2005). However, Blackboard is certainly much more expensive to maintain as it incurs a yearly cost of licensing. The benefits and risks of adopting an open-source platform vis-a-vis a commercial one have been explored elsewhere (Britain & Liber, 1999). Evaluations of both Moodle (Gower & Barr, 2005; Graf & List, 2005) and Blackboard (Hall, 2006) on their own have also been produced exploring all the relevant technical issues. However, there are not many studies directly comparing users’ experience and evaluation of Moodle and Blackboard. Most such studies have been conducted within the growing number of universities across the world using Blackboard but considering open-source alternatives such as Moodle. On the whole, these studies seem to show a preference of users towards Moodle. Bremer and Bryant (2005) present results of a survey of 14 students, 80% of which state an overall preference for Moodle over Blackboard. In a study involving 84 education students in Hong Kong, Kennedy (2005) reports that 45% of students preferred Moodle over Blackboard, although the majority (49%) had no preference. Bos, Munoz and Van Duzer (2005) also show mixed results on their survey of 35 students. Moodle was favoured by students for its ability to promote interaction with fellow students and with the instructor; whereas, Blackboard was preferred in terms of the organisation of instructional materials. In the study by Machado and Tao (2007), however, course material organisation and communication were rated higher in Moodle than in Blackboard. The overall evaluation of the 53 students surveyed tended towards Moodle, 71% of them finding it easier to use than Blackboard and 75% stating that they would prefer to use Moodle in the future as a replacement for Blackboard. In a recently published report by the University of Canterbury in New Zealand, of a study involving 750 respondents, 44.5% of the students judged Moodle to be better than Blackboard and 27.2% preferred Blackboard; whereas, 26.9% stated no preference between them (University of Canterbury, 2008). On the whole, these studies suggest students tend to prefer Moodle over Blackboard, although to varying degrees. As for more detailed explanations of students’ preferences, these studies present more disparate and less comparable results. Moreover, other than the study by Canterbury University, they have been confined to rather limited student samples and focused primarily on students’ perceptions and opinions of particular aspects of the platforms, making little if any connections with the overall impact of the LMSs on the enhancement of learning. In this paper, we try to extend the comparison between Blackboard and Moodle in terms of their potential contribution to higher levels of student engagement. The use of LMSs in the University of Minho The University of Minho (UM) has pioneered the use of LMSs in Portugal. After an initial experiment with ‘Easy’, an in-house co-development with a Brazilian university, UM settled for Blackboard in 2006 (Microsoft, 2007). It has gradually been promoting Blackboard among faculty, providing training and technical support via its SAPIA service (SAPIA stands for Serviço de Apoio Informático à Aprendizagem, and translates to IT Support Service for Learning). The adoption of the Bologna Declaration principles prompted widespread discussion of new teaching and learning approaches and methods among faculty, accelerating the take-up of the LMS. Active learning, cooperative group work and autonomous student work are also encouraged methods. Intent on supporting them, UM implemented a wi-fi connection in its two campuses and promotes the generalised use of Blackboard. 826 British Journal of Educational Technology Vol 42 No 5 2011 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. In parallel to the development of Easy and the purchase of Blackboard, many faculty members have begun to use Moodle on their own initiative as from about 2005. While Easy and Blackboard were at experimental stages, some departments (particularly in more technological areas) hosted Moodle instances within their Internet subdomains. Additionally, increasing numbers of students coming to UM, where they encounter Blackboard, have experienced Moodle in secondary school. UM is therefore an ideal study ground for the comparison of Blackboard and Moodle because most students have used either one or both platforms roughly between the school years 2005–06 and 2008–09. Data collection We surveyed all the students enrolled in UM during the school year 2008–09 (around 15 000), both at undergraduate and at postgraduate levels. The survey comprised four parts: (1) identification, (2) level of experience, (3) features used, and (4) satisfaction with Moodle and Blackboard. In the first section, biographical data was collected, as well as level and area of studies. The second part covered students’ level of experience with LMSs, including whether they had used LMSs in contexts other than UM (eg, in secondary school), and their self-perceived assessments of ease and frequency of use of both Moodle and Blackboard. The level of student satisfaction with LMSs has been shown to be affected, among other factors, by the level of computer skills and LMS experience (Drennan, Kennedy & Pisarski, 2005). Ease of use has been argued to impact the acceptance of technology by students and, consequently, the success of LMS implementation. If the technology is perceived as being complex, it may pose as a distraction to the learning process (Brown, 2002). Ease of use has also been shown to be a strong predictor of perceived usefulness and intention to use (Hussein, Aditiawarman & Mohamed, 2007). It also appears to be more important for recent users, the level of satisfaction of experienced users being less affected by ease of use (Drennan et al, 2005). In the third part, students were asked about which features of Moodle and Blackboard they use most frequently. This seeks to assess the depth to which the LMSs are being used and the level of engagement of students. In the final part of the survey, students were required to rate their satisfaction with Moodle and Blackboard on a number of features, including access to the platforms, organisation of course materials, navigation intuitiveness and communication tools. The degree to which they have felt the need to seek technical support was also queried. The survey was administered electronically, using the Qualtrics survey software (Provo State: Utah Country: USA) (http://www.qualtrics.com). Invitations to reply were sent by email during the month of April 2009. We had 876 questionnaires submitted, which corresponds to a response rate of nearly 6%. Although this rate is low, the size of our sample is very satisfactory, indeed the largest we have come across in studies comparing Blackboard and Moodle. Results and discussion Overall description The large majority of students (80%) are 18–27 years old, which fits the general age profile of undergraduate and master level students. The greater incidence of students between 19 and 23 (51%) is explained by the proportion of undergraduate students in the sample (72%). As for gender, 58% of the respondents were female, slightly over representing them in UM, where they account for 51.3% of students (student data for UM were collected from internal unpublished sources.). Student Perceptions of Blackboard and Moodle 827 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. Nearly three quarters of the respondents (72%) were undergraduates and 28% were postgraduate students attending Masters programmes (Figure 1). Postgraduates are also overrepresented as they make up for only 16.6% of UM students. Some of these attend courses given by the authors and may have been more predisposed to taking the survey than students with whom the authors were unfamiliar. The same explanation might apply to the sample distribution among areas. Although all areas of learning are covered (Figure 2), the larger groups of students were in Management or Economics (40%) or some kind of Engineering (31%). In the overall UM distribution, only about 20% of students attend Management and Economics programmes, and over 36% are enrolled in Engineering. Law and Social Science students are also slightly overrepresented, with 14% against a University proportion of 11.7%. On the contrary, Science students seem to be underrepresented (6% in the sample against 12% in UM), as are Arts and Humanities students (2% in the sample Figure 1: Distribution of students between Undergraduate and Postgraduate programmes. Figure 2: Distribution of students by study areas. 828 British Journal of Educational Technology Vol 42 No 5 2011 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. against nearly 6% in UM), Education (3% in the sample against 9.6% in UM) and Health Care students (3% in the sample against 4.5% in UM). Around 28% of all respondents had a ‘student-worker’ status, which means they might be less willing/able to come to classes and could benefit more from the distant learning functionalities of LMSs. Almost all students (98%) declared to be reasonably, quite or very comfortable with using IT. This reflects the fact that students nowadays arrive at university with good technological skills. It could also indicate some degree of self-selection bias. Students that are less comfortable with IT might have refrained from responding. The larger group of respondents (58%) had used both Blackboard and Moodle, and 29% had used only Blackboard. Only 5% had used Moodle alone, and another 8% had used neither (Figure 3). These results may not be representative of all students in UM, as explained above. In this instance, we feel our own students (who will have used both Moodle and Blackboard) might have been more influenced to take the survey. Few students had previous experience with these LMSs, with 88% of students using Blackboard and 76% of Moodle users never having used them before coming to UM. Those who had used Moodle had done so mostly in secondary school (10.8%). A further 5.3% encountered Moodle in another university, 3.7% in a professional training setting and 4.3% in other various contexts. Blackboard, on the other hand, had been previously used in other universities by 5.9% of students and, to a minor extent, in other settings (Figure 4). A large proportion of students (44.7%) declared that they did not use other collaborative platforms. Almost a quarter (22.9%), however, use googlegroups, and 10.6% use yahoogroups. The most cited reasons for using these alternatives include ‘my colleagues use it’ (18.1%), ‘they have better functionalities for team work’ (16.77%), ‘I already used it’ (16.1%) and, curiously, ‘so that the instructor or other colleagues don’t interfere in the team’s work’ (17%). Also revealing is that 15.2% of students using alternative collaborative platforms do it because ‘it’s what the instructor uses in their course’. This means there are a number of faculty using alternative systems despite the fact that Blackboard is available to them. Perhaps they are not entirely satisfied with it, or they might be using these alternatives before, and are now unwilling to transfer to a new system. Figure 3: Distribution of students by LMS used. Student Perceptions of Blackboard and Moodle 829 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. Levels of engagement As for the level of engagement to which the LMSs are used, as expected, the vast majority of students seem to experience no higher than Francis and Raftery’s (2005) Mode 1. There is a very clear pattern whereby the frequency of use decreases as the LMS functionality becomes more ‘engaging’. Figures 4 and 5 illustrate this most vividly. Indeed, students using Blackboard (Figure 5) very often ‘download course materials’ (78.8%), ‘check course announcements’ (63.4%) and ‘check course grades’ (47%); but most will only sometimes or rarely ‘check the course syllabus and schedule’ (36.7% and 34.6%, respectively) or ‘submit assignments’ (31.4% and 28.5%); nearly half have never used Blackboard to ‘send emails to faculty or colleagues’ (46%) nor to ‘take quiz tests’ (48.3%); and the vast majority have never engaged in ‘participating in forum discussions’ (72.4%), ‘sharing files with colleagues’ (74.2%) or ‘participating in course chat-room or virtual classroom’ (89.9%). Likewise, in their experience with Moodle (Figure 6), students tend to ‘download course materials’ (67.2%), ‘check course announcements’ (47.5%) more often than they ‘submit assignments’ (41% do it very often and 35.4% do it sometimes) and ‘check course grades’ (38.7% very often; 34.5% sometimes). Most students will only ‘check the course syllabus and schedule’ (33.4% sometimes; 40% rarely), ‘take quiz tests’ (26.5% sometimes; 21.4% rarely) or ‘check faculty and colleagues’ contacts’ (28% sometimes; 35.2% rarely) sometimes or rarely. Nearly half never ‘send emails to faculty or colleagues’ (48.2%) or ‘take course surveys’ (44.6%), and a greater proportion never engage in ‘participating in forum discussions’ (60.4%), ‘taking an online appointment with faculty’ (60.8%), ‘sharing files with colleagues’ (73.9%) or ‘participating in course chatroom or virtual classroom’ (86.7%). Despite the same general trend, Moodle users more frequently use some of the more active features, such as submitting assignments and taking quiz tests. To test whether Moodle is being No Yes, in secundary school Yes, in another University Yes, in professional tranning Yes, in another setting 100 95 90 85 80 75 70 65 60 55 50 45 40 35 30 25 20 15 10 5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Blackboard Moodle 88.01 % 76.02 % 2.7 % 10.77 % 5.85 % 5.28 % 1.95 % 3.66 % 1.5 % 4.27 % Figure 4: Previous experience of students with the LMSs: “Have you used Blackboard/Moodle before coming to UM?” (percentage of students in each situation). 830 British Journal of Educational Technology Vol 42 No 5 2011 © 2010 The Authors. British Journal of Educational Technology © 2010 Becta. used to a higher level of engagement than Blackboard, we checked for statistically significant differences between the means of the frequencies using the Mann–Whitney test, which is specifically indicated for analysing ranked ordinal type data and data that does not follow a normal distribution (Conover, 1980). The results show (Table 1) that there are indeed statistically significant differences across most features. In a scale from 1 (very often) to 4 (never), students report being more involved in ‘checking course announcements’, ‘checking the course syllabus and schedule’, ‘downloading course materials’, ‘checking course grades’ and ‘checking faculty and colleagues’ contacts’ when they are using Blackboard than when they use Moodle. Inversely, they are more likely to engage in ‘submitting assignments’, ‘taking quiz tests’, ‘taking course surveys’, and ‘participating in forum discussions’ or ‘participating in course chat-room or virtual classroom’ when they are using Moodle. It seems that, to some extent, the utilisation students make of Moodle encompasses higher engagement than that of Blackboard. This suggests that Moodle may be being utilised closer to Francis and Raftery’s (2005) Mode 2 as students make more use of communication and collaboration tools. The explanations we find for these differences are that Moodle is used willingly (indeed wilfully) by only a few faculty, most of which are in the Information Systems Department (ISD). These instructors may be more motivated and capable of using LMSs to a higher level of engagement. Blackboard, on the other hand, is being strongly encouraged among all faculty members, and a good part of them will use it only tentatively, exploring the more basic functionalities. D ow nl oa d co ur se m at er ia ls